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ABSTRACT 



A correlation surface is derived by phase correlating 
two pictures selectively displaced in the X and Y direc- 
tions. The illustrative surface shows a large peak at zero 
displacement, corresponding to a stationary back- 
groimd, and a fairly large peak corresponding to a mov- 
ing object, the X, Y position of the peak indicating the 
magnitude (pixels per field period) and direction of the 
motion vector. A set of motion vectors is thus deter- 
mined and testing is then carried out, on a pixel by pixel 
basis or pixel block by pixel block basis, to determine 
which of the motion vectors gives the best match in 
deriving the second picture from the first. The motion 
vector thus assigned may be used in temporal interpola- 
tion of the pictures. 

27 Oaims, 6 Drawing Sheets 
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The resulting phase correlation array can be thought 

TV PICTURE MOTION VECTOR MEASUREMENT of as a surface whose height at a particular point (x,y) is 

BY CORRELATION OF PICTURES proportional to how well the two images correlate 

when the relative displacement between them is (x,y). 

1.0 FIELD OF THE INVENTION 5 In the case of a simple shift between the two pictures, 

There are many picture processing apphcadons in ^« correlation surface would be a delta fimction cen- 

which knowledge of the speed and direction of move- ^^<^ shift vector. The idea is that there will be 

ment of all parts of the TV picture would be very use- a peak in this surfeced for each dommant motion vector 

fnl. These applications include standards converters, '^ ^^ scene. Measuring these motion vectors involves 

noise reducers, bandwidth reduction schemes, and oth- ^^^E for large peaks in the surface. The relative 

ers where any sort of temporal interpolation is required. ^^S^^ of the peaks will reflect the relative sizes of the 

Even information concerning pans and the movement moving objects. The mam novel feature of this method 

of the larger objects in the scene would be very useful. « ^ look for several peaks rather than just one. thereby 

allowing the detection of many velocities in one opera- 

1.1 PRIOR ART 13 tion. 

Various techniques have been used to measure mo- Jo measure the motion vectors to sub-pixel accuracy 
tion m TV pictures. The most promising of these ap- ^. necessary to perform some interpolation on the 

peared to be a method based on phase correlation, correlation surface. 

which has been shown to be capable of measuring large ^ order to measure as many of the velocities present 

movements of the complete scene to sub-pixel accuracy. m a scene as possible, it helps to divide the picture up 

(PEARSON, J. J. HINES. D. C, GOLOSMAN, S,, into blocks, rather than to perform correlations on 

KUGLIN, C. D. 1977 Video-rate Image Correlation whole pictures. This is because the number of individual 

Processor. S.Pi E. Vol. 119. Application of Digital peaks tiiat can be detected accurately is limited by noise 

Image Processing (lOCC 1977)). ,5 '° ^^'out 3 peaks per block. In addition it is only possible 

to resolve velocity peaks if they are separated by a shift 

1.2 THE INVENTION vector greater than about one pixel per field period. The 

The object of the present invention is to extend the block size would be large compared with the largest 

known technique to measure the motion vectors in a shifts that are expected, as the technique cannot corre- 

scene containing many objects moving in different di- late objects that move between blocks. A size of 64 by 

rections and speeds. 64 pixels is convenient. If on the other hand, it is only 

The invention is defined with particularity in the necessary to measure the few most significant motion 

appended claims. The preferred practice of the inven- vectors, then it may be as well to transform whole pic- 

tion is explained in section 2 below. tures, or at least very large blocks. It should be remem- 

^ « « * * ^^r^T^^^ . -r^^^r^T^^^T^ ^crcd that this division into blocks is only made for the 

2.0 BASIC PHASE CORRELATION TECHNIQUE 35 ^^^^^^ measuring motion vectors; the vecotrs are 

The process of calculating motion vectors for every still assigned on a pixel-by-pixel basis, 
pixel m tiie picmre is broktm down into two stages. The TWO-VECTOR ASSIGNMENT 

first stage mvolves correlatmg two successive pictures * oii^vj^i i mrw ^ cv^xv-nv rt»3oxwi^i«i-i^ i 

(or fields, depending on the exact application) to deter- ^ The first stage of the process gives us a set of motion 
mine the principal motion vectors present in the scene. vectors that are present in the scene, but tells us nothing 
The second stage attempts to assign one of these vectors about which parts of the scene are moving with which 
to every pixeL For some pixels it may be impossible to vector. The second stage involves 'trying out* each« 
assign a motion vector, for example if the pixel corre- possible vector on every pixel, and assigning the vector 
sponds to a very small object or to uncovered back- 45 that gives the best *fit'. In situations where motion infor- 
ground. mation is required on a block-by-block basis (for exam- 

2.1 STAGE ONE-VECTOR MEASUREMENT ple a block based bandwidth compres^on system^ 

M^^x^M^ * would be no need to assign a vector to each pixel, and 

In the first stage of the process, 2-dimensional Fast the assignment would be done on a block basis. 
Fourier Transforms (FFTs) of the luminance compo- 59 For every detected vector, an *error surface* is 
nents of two successive pictures are calculated. Then formed by calculating the modulus difference between 
for each spatial frequency component in the transforms, the two input pictures when shifted by the motion vec- 
a unit length vector is calculated whose phase angle is tor imder consideration. Areas of the two picnires that 
equal to the difference in the phase s of this frequency in match well will produce low errors, and it is reasonable 
the two pictures. A reverse FFT is performed on the 55 to assume that these areas correspond to a moving ob- 
resulting complex array, which produces an array of ject with this particular motion vector. It is preferable 
real numbers giving the correlation between the two to perform some sort of spatial filtering on the error 
pictures. Mathematically, if Gj and G2 are the discrete surface in order to reduce the effects of noise. 
2-dimensional Fourier transforms of the two successive if the motion vectors are bemg measured to sub-pixel 
images, then the complex array Z is calculated at every gQ accuracy, it will be necessary to use some sort of spatial 
spatial frequency (m,n) using interpolator when calculating the error surface. 

> Once all the motion vectors measured in the &st 

Gi(m.n)(him,n)* Stage have been tried, each pixel is assigned the motion 

iCi(m,n)G2(m,n)*| vector which gave the smallest error value. A threshold 

65 error level could be set, defining the maximum accept- 
and the phase correlation is given by the inverse Fou- able error level. Pixels whose errors are above this level 
rier transform of Z, which will only have real compo- with every trial vector could be flagged as 'motion 
nents. unknown'. Such pixels would probably correspond to 
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areas of erratic motion, covered or uncovered back- denoted X-V and Y-V and measured in pixels/field 

ground. period. 

If the motion vectors were measured by transforming FIG. 1 shows the correlation surface for a stationary 

the input pictures in blocks rather than as a whole, then picture using a SIN (x)/x interpolator, 
it is only worth trying vectors measured in the neigh- 5 FIG. 2 shows the surface for a pair of 10 pixels per 

bourhood of the pixel under consideration. The chances field period again using a SIN (x)/z interpolator, 

are that most pixels could have the correct vector as- FIG. 3 shows the surface for a stationary picture 

signed to them merely by considering the vectors mea- using a smoother interpolator. 

sured in the block containing the pixel. Ideally the vec- FIG. 4 corresponds to FIG. 2 but using the smoother 
tors measured in the immediately surroimded blocks 10 interpolator. 

should be tried as well, particularly for pixels near the FIG. 5 illustrates windowing of a phase array prior to 

edge of blocks. This is important in situations where a reverse transformation. 

small part of a moving object intrudes an adjacent FIG. 6 shows the correlation surface for a pair of 10 

block. pixels per field but with a "guard band" on the input 

" '^^A^S'' ^"^^^ -> «'"«P°"«1^ ^° F^G. 6 bu, further with a 

raised cosine aperture on the input picture. 

There are two basic methods of motion estimation FIG. 8 shows the correlation surface for an object 

that have been used other than phase correlation, moving over a stationary background, 

namely spatiotemporal gradient techniques and optimal 20 RG. 9 shows the correlation surface for a moving 

matching. gate in a "Voit" sequence referred to below. 

Both of these techniques fail for large movements, FIG. 10 illustrates how a velocity vector can be as- 

and although applying them recursively from picture to signed in performing a motion compensated intcrpola- 

picture helps, they still fail for rapidly accelerating tion. 

objects and can take a significant length of time to re- 25 FIG. 11 shows how both a large and a small vector 

cover after shot changes. It is often necessary to divide can fit one point. 

the picture into small blocks and assign motion vectors The purpose of the computer simulations that have so 

on a block-by-block basis, which means that the edges far been performed is to find out how well the phase 

of moving objects will often have incorrect velocity correlation technique really works. We have not limited 

vectors assigned to them. 30 the simulations to algorithms that would be easy to 

Phase correlation overcomes the problems with these implement in hardware; the intention was to find out 

more 'traditional' techniques. For small movements, all how good the best algorithm is. The subject of further 

three techniques can be shown to be roughly equivalent simulations will be to find out how much the 'ideal* 

but, for large movements, phase correlation comes into algorithm can be simplified in order to make it suitable 

its own, 35 for a real time hardware implementation for a given 

The act of performing correlations between large application, 
sections of two successive pictures is equivalent to using 

an optimal matching technique with a very large num- ^'^ INVESTIGATION OF VECTOR 

ber of trial vectors. The quickest way to perform such MEASUREMENT 

correlations is by calculating the FFTs of both pictures, 40 The first phase of the investigations examined the 

multiplying them together and untransforming the re- Vector measurement* stage of the algorithm. The aim of 

suiting array. Once this approach has been adopted, it is this phase was to investigate the accuracy of the phase 

very little extra effort to perform a phase correlation. correlation technique, and see how it depended on the 

This has the advantage of making the technique inmiune relative shift size, number of moving objects and object 

to brightness changes in the scene, and tends to produce 45 size, amount of noise, and so on. 

sharper peaks in the correlation surface. A phase corre- ^ ^ ^ 

lation can be performed with simpler hardware than ^'^'^ MEASUREMENT ACCURACY FOR SIMPLE 

would be needed for a cross-correlation, since it is possi- PANS 

ble to quantize the unit amplitude vectors in Z quite Initial investigations dealt with simple pans. Two 

coarsely. 50 picture portions were correlated, both taken from the 

This technique has the intrinsic advantage that it same larger picture but with slightly different locations, 

detects the dominant motion vectors in a scene. These Both picture sessions were 64 pixels square, luminance 

would usually correspond to large areas of background only, and were taken from a portion of the test slide 

and large moving objects. These are the sort of things POND which contained a reasonable amount of detail, 

that the human eye can easily follow, and hence they 55 FIG. 1 shows the correlation surface obtained when 

are the most important parts of a scene to process cor- two identical picture portions were correlated. The 

rcctly. phase array Z was padded out with zeroes to form a 128 

-K n npcr-PftrrToxi ni? rsx> awtm^^c axtt^ ^^8 array prior to performing the inverse FFT; this 

60 with a sm (x)/x unpulse response. This is the cause of 
The invention has been investigated" in detail by the 'ringing' visible around the peak. FIG. 2 was pro- 
means of computer simulation of the phase correlation duced in the same way but with a horizontal shift of ten 
technique and will be described in more detail with pixels between the two picture portions. The location of 
reference to the accompanying drawings, in which: the peak has moved, and the use of quadratic interpola- 
FIGS. 1 to 4 and 6 to 9 show various correlation 65 tion (applied independently in the x and y directions) 
surfaces, as detailed more fully below. In all these draw- gives the peak location as (9.99,0.02). Hence the shift 
ings the Z axis shows correlation peaks, while the X and has been measured to an accuracy of a few hundredths 
Y axes show X velocity and Y velocity respectively, of a pixel in ten pixels. The height of the peak has dimin- 
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ished and noise has been introduced: both effects can be It is possible that some improvement in signal-to- 
attributed to the revealed and obscured material at the noise ratio may be obtained by using 'adaptive win- 
edges of the picture, dows'. In the basic phase correlation technique, the 
Similar experiments were tried with other shift val- modulus of all the numbers of the phase array Z is set to 
ues. The technique was found to be capable of accu- 5 unity. This means that the information in frequencies of 
rately measuring shifts up to the instrumental limit of amplitude carries more weight than perhaps it 
half the picture size (32 pixels m this case). Shifts any should, given that the signal-to-noise ratio of these corn- 
larger than this simply produce peaks at locations corre- ponents will be low. In comparison, no such normaliza- 
spondmg to smaller shifts in the opposite direction (e.g., ^qj^ performing a cross-correlation/ 
a shift of -h34 pixels appears as -30). Fractional shifts 10 ^^^^ correlation peaks are broader 
were mcasui«i to an acci^y of about 0.02 pixel using ^ heights become scene^lependent, A com- 
thc mtcrpolation mcrthod d«cribcd above. Simp y usmg ^^^^ ^ correlation and cross-corrcla- 
quadratic mterpoiatoon without precedmg sm (x)x inter- ^^^^ ^ ^^^^ ^^^^ 
polation reduced the measurement accuracy to about ^ normalizing higher ampUtude components 
' and reducing the ampUtude of low ones. These ideas 
3.1.2 THE EFFECT OF DIFFERENT wiH be investigated in the next stage of the simulation 
INTERPOLATION FILTER WINDOWS work. 

The ringing introduced by using the sin (x)/x interpo- 3,1,3 THE EFFECT OF DIFFERENT INPUT 
later produces spurious peaks which can sometimes 20 PICTURE WINDOWS 

mask peaks produced by small moving objects. The , , , , . 

ringing can be reduced by using a different interpolator. " ^^n suggested that some sort of padding with 
The interpolator can be changed by windowing the wmdowmg ought to be performed on the 

phase array 2(m,n) prior to performing the reverse "^P^* pictures before they are transformed. The reason- 
transform. The effects of various windows were investi- 25 iag behind this is that we would introduce unwanted 
gated. The windows were all of the form tO^flg to correlate (say) the left and right hand 

edges when calculating the correlation for a shift with a 

horizontal component. 

a + (1 - a)cQs(f//o • TT/i) f <-fo Experiments conducted using a guard band of zeroes 

0 f>fo 30 (but no other windowing) showed that although a small 

reduction in noise could be produced, a spurious peak at 
where f is the horizontal or vertical frequency, and fo is zero velocity is formed, FIG. 6 shows the same correla- 
tbe highest frequency supported in the original array tion as FIG. 2 but with the input pictures padded with 
before any zeroes are inserted, FIG. 5 shows the win- zeroes to form pictures that were 128 pixels square. The 
dowing arrangement graphically. 35 spurious peak can be clearly seen, and is probably due to 

The solid line shows a simple window with fi-equen- the sharp edge between the picture portion and the zero 
cies 0 to fff supported in the array Z and zeroes inserted 'guard band* correlating in the two pictures, 
ftom fo to 2fo to enable interpolation of intermediate The effect of applying a raised-cosine window (of the 
points. This gives the sin (x)/x impulse response with form used on the phase array Z) to the mput picture 
sharp peaks but "ringing". The broken line shows a 40 portions was also investigated. When applied in con- 
Hamming-like window leading to poorly resolved main junction with a 'guard band*, the spurious peak at zero 
peaks but smaller subsidiary peaks. This one dimen- velocity was removed (confirming the idea that it was 
sional windowing function was appUed both horizon- caused by the sharp join between picture and guard 
taUy and vertically. A value of zero for V gives the . band). A further improvement in signal-to-noise ratio 
rectangular wmdow that produced FIGS. 1 and 2. and 43 obtained- FIG. 7 shows the correlation surface for 
a value of 0.54 would give a Hammmg window. Expen- ^^^^^ ^ions as FIG. 2, but with a guard 

ments showed that a value of about 0.8 gave a reason- ^ -^^^ ^ raised-cosine aperture on the 

able compromise between heighte of spurious peaks and .^^^^ ^^^^^ signal-to-noise ratio has fiirther 
sharpness of each peak. FIGS. 3 and 4 correspond to r j j *i. 1 * 1 v 

FIGS. 1 and 2 but with an interpoUtion fUtL with 50 «?P^oved andthe spunous peak at zero velocity has 
a^as. The filter has the effect of reducing aU the peak d^^PPff^d. The computaUonal penalty for this ap- 
heights and broadening them sUghtly, but does not af- P/^^^^ " transform deals with smaller areas of 

feet the interpolated peak locations. thepicture. . . ^ ^ ^ . . 

The number of situations where the spurious peaks . mvestigations showed that an unprovemcnt m 
cause problems is smaU. and as the type of interpolator 55 signal-to-noise ratio can be obtamed by wmdowmg the 
that would be required to remove all the spurious peaks "^P^^ picture with a raised-cosine aperture and sur- 
results in unacceptable broadening of the main peaks, roundmg with zeroes. However this represents a signifi- 
then it may not be worth usmg anything other than a increase in the processing power required, and the 

simple rectangular window. improvement is signal-to-noise ratio is quite small (less 

Another way of avtriding the spurious peaks would 60 * factor of two). Padding the input picture with 
be to perform the peak hunting operation on the unin- zeroes without performing any other windowing is 
terpolated surface. Once a peak had been located, the detrimental, as a spurious peak is introduced. Use of a 
height of the correlation surface at half-integral shift raised-cosine window without padding with zeroes 
values could be calculated using an interpolator with a reduces the noise slightly, but would mean that overlap- 
sin (x)/x type of response. This has the further advan- 65 ping blocks would have to be used in order to detect 
tage that less points have to be searched to locate max- motion throughout the picture. Hence in many applica- 
ima. This approach will be the subject of further com- tions it is probably not worth bothering with any sort of 
puter simulations. windowing. 
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1 1 A TMP PFPPPT OK Nrni^F P"^^* Object's motion vector was 

3.1.4 1H£ hi^thi^i JNUlbii accurately detected for objects as small as two pixels 
The way in which noise on the input pictures affects square, which was quite surprising, especially consider- 

the results of a phase correlation was investigated. ing that the object had moved more than twice its own 

Noisy pictures were generated by adding random val- 5 length. 

lies (with a given peak value) to the luminance level at Investigations with two objects moving over a sta- 
cach pixel. A correlation was performed between two tionary background showed that 3 peaks (1 for the 
pictures shifted by ten pixels (as used to generate FIG. background and the 2 for the objects) could usually be 
2), with different levels of noise. It was found that add- detected. In some cases though, the velocity peaks were 
ing noise with a peak value 20 dB below peak white 10 becoming obscured by noise (possibly due to uncovered 
level reduced the height of the correlation peak by a background) and spurious peaks from the sin (x)/x in- 
factor of 2.4, without significantly changing the general terpolator. The amount of noise could probably be re- 
level of noise on the surface. The measurement error duced by the use of windows on the input picture por- 
increased to about 0.08 pixel. Increasing the amount of tlons and some form of adaptive windowing on the 
noise to 10 dB below peak white level caused a further 15 phrase array, as discussed earlier. The spurious peaks 
reduction of a factor 2.2, but the peak was still detect- from the interpolator may not present a problem if the 
able. The measurement error rose to about 0,3 pixel. peak hunting process is carried out on the uninter- 
This suggests that noisy pictures do not present a signifl- polated array. These points will be the subject of further 
cant problem to this part of the motion measurement mvestigations. 

process. It is possible that the noise immunity could be 20 As a final test, the correlation surface for two succes- 

increased by some 'adaptive windowing' techxiique on sive fields from a real sequence was generated. A por- 

the phase array, as discussed in section 3.1.2, tion of the 'VOIT' sequence that showed the gate mov- 

A possible way of reducing the noise on the correla- ing fairly rapidly over a roughly stationary background 

tion surface would be to perform some kind of temporal was chosen. FIG. 9 shows the correlation surface ob- 

filtering operation. For example, a first order recursive 25 tained. The two large peaks correspond to velocities of 

temporal filter applied to the correlation surface of each (0.03,-0.42) and (2.38-0.25). Measurements (using a 

block would enhance stationary or slowly changing rulerl) suggested that the speed of the gate was about 

peaks, while reducing the level of any noise peaks. The 2.40 pixels per field period, which agrees very well with 

disadvantage of this method is that velocity peaks one of the values obtained by correlation. Since the 

caused by rapidly accelerating objects would be attenu- 30 correlation was between two successive fields, a verti- 

ated. It should be possible to choose a filter which gives cal shift of half a pixel would be expected, which agrees 

good attenuation of noise without unduly attenuating reasonably well with the vertical shifts detected. The 

genuine moving peaks. It would be advantageous to heights of the two large peaks are roughly the same, 

detect shot changes and inhibit the temporal filtering reflecting the fact that the gate and the background 

action immediately after a change. 35 occupy areas of roughly the same size. 

3.1.5 SCENES WITH MORE THAN ONE MOTION 3.1.6 SUMMARY OF MOTION VECTOR 

VECTOR DETECTION INVESTIGATIONS 

Once the technique had been shown to work well for The experiments described above showed that - the 
pans, pictures with an object moving over a back- 40 phase correlation technique does indeed work as well as 
ground were investigated. The aim of these investiga- claimed in the reference cited above. For pans and one 
tions was to see if the technique could detect accurately moving object typical vector measurement accuracies 
several moving objects by producing several peaks. of 0.02 pixel can be obtained by first interpolating the 

A portion of a picture ('Dick and JeanO 64 pixels correlation surface by a factor of two in both directions 
square was extracted, and a 32 by 32 pixel portion from 45 (using an interpolator with an impulse response roughly 
another picture (the blackboard cross from Testcard of the form sin (x)/x), then fitting a quadratic to the 
F') was inserted into the middle of this picture. The interpolated points either side of the maximum (for x 
edges of the inserted pictures were *blended' with the and y separately). An accuracy of about 0.2 pixel can be 
background picture over a distance of 3 pixels so as not obtained by simply fitting a quadratic to the uninter- 
to produce any artificially high frequencies m the result- 50 polated correlation surface. The accuracy of the tech- 
ing picture. A second similar picture was formed, but nique was largely unaffected by noise, and it was possi- 
with the inserted picture portion shifted three pixels to ble to detect very small objects, 
the left and three down. In order to be able to measure the velocities of more 

FIG. 8 shows the resulting correlation surface. The than 2 or 3 objects in the picture area being correlated, 
interpolated peak locations (calculated by fitting a qua- 55 it may be advantageous to make some modifications to 
dratic to the sm (x)/x interpolated data as before) were the basic technique. One possible modification would be 
accurate to 0.01 pixel. The relative heights of the peaks to window the input picture portion with a raised- 
refiect the relative areas of moving objects and back- cosine function and possibly pad it with zeroes, which 
ground, although the ratio of the heights is less than the can reduce the correlation surface noise. It may also be 
ratio of the object areas (1.6 compared to 4). The noise 60 possible to reduce the noise by applying a sort of adapt- 
around the peaks is due to the uncovered and revealed ive window to the phase array, in such a way that small 
background around the inserted picture portion. amplitude components carry less weight. The problems 

As the phase correlation technique had no problems experienced with spurious peaks produced by the inter- 
accurately measuring this motion, some more severe polation process could probably be avoided by carrying 
experiments were tried. The aim of these was to see 65 out the peak hunting process on the unintcrpolated 
how small a moving object can be, and still be detected. array. Further simulation work is required to determine 
A picture portion of various sizes was 'moved* with a which of these improvement are worth incorporating 
shift vector of (5,5) over a background picture, which for particular applications. 
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1 -7 novrrcTTr' a -rrnxi r^c vcr-rnu implcmcntatioiu pixels with large errors would be 

3.2 lNVE5TlG^l^iNOF VECTOR investigated further, and some account of uncovered 

background would be made. 
The aim of this part of the investigation was to simu- The luminance value of each pixel in the output pic- 
late the second stage of the motion measurement pro- 5 ture was calculated by averaging the values in the adja- 
CC33. namely assigning the principal motion vectors to cent two fields, at locations displaced by the motion 
particular pixels. The first problem that had to be dealt vector for the pixel, FIG. 10 illustrates this idea, 
with was to find a way of showing how weU the vector piQ. lo shows a previous filed P, a next field N and 
measurement and alignment had been carried out It ^he field G being generated. The luminance value at the 
was decided to do Uns by usmg the mouon vectors to lo po^^t in field G is the average of the values at points x 

"^^J^^^^^^^f P^^'^ry^?*!^^ and X in fields P and N where the points x and z are 

pictures. This is not only a strmgent test of the method displaced from y by ± V/2 and V is Se velocity vector 
but also shows w^ sort of rcsdte codd be obtomed if ^ ' ^his vector will have been chosen 

KS^n^'^S'^fi?^^^ to iSumise^be llinance difference between points x 

film, by n«dang up mtcrmcduite pictur^. 15 ^ filtering). 

A computer program was developed that could gen- ^ a^^auai miwm^/. 

erate the odd fields of a sequence by temporal interpola- 3.2.2 RESULTS 

tion between the even fields (or vice versa). The pro- „ . . - , . . . . ,^ 

gram allowed the user to change various parameters ^sing method oudmed above, the even fields 

- firom the *VOrr' sequence were generated from the odd 



compensated interpola- 



associated with the method, such as the size of the input 20 , ,7^*^ scqucnccwert 

picture, the size of the blocks on which correlations P«fonmng a motion 
were performed, the number of vectors extracted per 

block, and so on. ha!dal investigations performed without any spatial" 

filtering on the error surface showed that incorrect 

3.2.1 DETAILS OF THE METHOD USED vectors were often assi^ed to pixels due to noise. The 

All these investigations were done using the 'basic* use of the spatial filter discussed above cured this prob- 

phaae correlation technique (as described in section 2), The penalty for using a spatial filter b that the 

with simple square windows on the picture portions and background immediately surrounding a moving object 

on the array Z. The correlation surface was interpo- occasionally gets 'pulled along' with the object. This 

lated by padding with zeroes, as discussed earlier. 3Q problem can probably be cured by changing the filter 

In order to see the technique working at its best, the type. In the case of revealed or obscured background 

correlations were carried out on blocks (64 pixels by 32 there is a fundamental problem: since the picture infor- 

lines) rather than on the whole picture, to measure as madon only exists in one of the two input pictures, no 

many motion vectors as possible. A maximum of 3. vec- motion vector can be assigned to it This problem* Te=-^ 

tors was measured i>er block. The menu of trial vectors 3 ^ quires a difference approach, and is discussed briefiy 

for a given pixel consisted of vectors measured in the later. 

block containing the pixel, as well as the immediately Once spatial filtering of the error surface was in- 
adjacent blocks. Thus for a pixel in a block in the middle eluded, the interpolated pictures started to look quite 
of the pkture, a n iaximu m of 27 vectors would be tried. presentable, and only one main problem remained. Most 
In order to assign a motion vector to a pixel, an 'error ^ parts of the interpolated picture appeared to be correct, 
surface' was calculated for each vector, as described in except for the occasional disappearance of lower parts 
section 2.2. Initial work used a very simple algorithm to of the car's radiator and sections of the moving gate 
interpolate the input pictures so that non-interger vec- ^posts'. This problem was found to be due to large mo- 
tor lengths could be dealt with. This algoritiim simply tion vectors being appUed to slowly moving areas in 
mvolvedtaldngaweightedsumofthevaluesofthefour cases where both large and small vectors would be 
nearest pixels; The weights were chosen such that when valid. FIG. 11 illustrates this problem in tiie case of the 
the pomt being interpolated coind^^^ car** radiator. This represents a statiomu^ object O 
of either of the four pixels, the mtcrpolated va^ne was a stationary miiform background. Botii Vj and 
1^"^ ""^ T 1?"^^: investigations V: are possible motion vectors for tiie point P in the 
I^r^^^^^ ^^a^^Ibwc ^^l^ 50 field G if the background U uniform. For example the 
cub^c splme fit HOU, H. S., ANDREWS, H. C 1978. ^^^^^ q be a silver surromid at the bottom of a 

"^BTr^^VV^^^ ca^radiatorSainstabackground provided byther^t 

'^^^l^J'^^^^ atoritself^d the black area MOW 

witiiasimplespatialfilterwithanap^eoftheform T^^c^t^^^l^^ 

lished July 1986. The reasoning behind the gate's disap- 
pearance is slightly more subtie. The gate is a periodic 
l/(dx + dy+ \) dx,dy < = 2 Structure which moves about 4.7 pixels horizontally in a 

0 dx,dy > 2 picture period, and repeats itself spatially about every 

60 14 pixels. This means that there are two valid motion 
where dx and dy are the absolute horizontal and verti- vectors for the gate, namely -h4.7 and —9.4 pixels per 
cal distances (in pixels) from the point in the error array picture period (disregarding the effect of the edge of the 
under consideration. This type of filter was used largely gate). If the incorrect motion vector is chosen, the gate 
because it is easy to implement, and could be refined. A 'breaks up' in the interpolated picture. 
difTerent form of filtering, such as median filtering, may 65 This problem was alleviated by multiplying the error 
prove to give better results. surface for each vector by a function that increased 

A motion vector was assigned to every pixel; there with increasing vector length. This means that when 
was no upper limit set for the acceptable error. In an two vectors give roughly the same match error, the 
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short of the two vectors will be assigaed. The weighting modification that was necessary was the incorporation 

function used was of the form of a weighting factor to bias the vector assignment 

towards small vectors. 

^3 There are several refinements that could be incorpo- 

• 10 + 1 ^ rated in the vector assignment algorithm, including 

improved filtering on the error surface (maybe a differ- 
ent sort of spatial filter, or even a temporal filter). There 

where v is the length of the motion vector and Vois the is also some scope for 'fine tuning' some parameters. In 

velocity at which the weighting function becomes 10. addition, the improvements suggested in section 3.1.6 

This function wais arrived at in a very arbitrary manner. 10 for the vector measurement algorithm could be incor- 

A value for Vo of 5 pixels per picture period was enough porated and these may improve the performance of the 

to persuade the program that the gate was in fact mov- technique as a whole. These improvements would be 

mg to the nght and not to the left. Investigations with particularly important if the motion vectors are mea- 

other picture sequenc^ need to be (^rried out to see sured by transforming the picture as one large block 

how generaUy apphcable this *fiddle factor* is. 15 .^jher than as many smaller blocks. Another usefiil 

Once this feature had been mcoiporated very pre- experiment would be to try to incorporate an ^uncov- 

sentebie output pictoes were produced The output ered backgromid' algorithm, 
pictures were shghtly soft, partially due to insufficient 

vertical detail being available in the input pictures (as 4.0 HARDWARE IMPLEMENTATION 

ti^i^alS^^^^^ " .Tliedetai^ofahardwareimpl^^^^^ 

tors^vith non^teger components. Kpatial i^terp^- TnnTn^S^^^ "^"""T 
lator was upgraded to one based on fitting a cubic ^P^^^^^^^ the motion mformanon was to be used 
spline, resulting in sharper output pictures at the ex- !°' ^^^.^^f^' ^^^'^ ^^^^ 5^°*^^^! pomts that would 
pense of a significant increase in the processing time 23 applicable to most implementations, and some of 
required. discussed below. In the followmg discussions, 

On examining a field from tiie VOIT sequence that ^ sampling rate of 13.5M Hz is assumed, 
was interpolated using the fuUy refined method, it is 4.1 VECTOR MEASUREMENT HARDWARE 
found that it looked surprisingly good, especially con- _ 

sidering the lack of an 'undercovered background' algo- 30 "^^^^^^ measurement part of the hardware would 

rithm. ^® based around a circuit that could perform one dimen- 

The only remaining problems (apart from undercov- ^ ^^^^ ^ circuit could be 

ered background) are that the motion vector of parts of muitipUers (or PROMs) and adders to 

the gate are sometimes assigned incorrectly (despite the implement one FFT ^butterfl/ operation. The data 
fiddle factor described earlier), and the spatial filter on 35 ^®"^g transformed would be held in RAMs, and a 
the error surface can cause the edges of objects (or PROM would generate the appropriate sequence of 
adjacent background) to appear slightly corrupted. adresses to operate the butterfly between appropriate 
Further minor refinements to the algorithm should be ^^AM addresses. Another PROM would hold the 'twid- 
able to cure these problems. die factors*. By usmg a pipelined design, such a circuit 

Areas of uncovered background could be detected by 40 would t ake n .(log2 n) clock pulses to do an n point 
examining the arrangement of motion vectors m the complex FFT. Such a circuit would consist of 8 multi- 
picture. There must be some obscured or revealed back- pliers, 6 adders, a selection of PROMs holding twiddle 
ground at the boundary between regions with different factors, and 4 RAMs, each holding a copy of the array 
vector components normal to the boundary. In order to being transformed. The physical size of such a circuit 
work out what picture information belongs in such 45 would depend on the number of bits required in the 
areas m an interpolated picture, more than two input calculation (which has yet to be determined), but it 
pictures need to be examined. would probably fit on 1 or 2 4U boards. 

Although these mvestigations involved dividing the A. controlling board could hold the picture portion 
input picture up into many blocks to perform the corre- (or the whole picture) being transformed, and pass it 
lation, this may not be necessary for many applications. 50 (one Hne or column at a time) to the FFT board. If the 
A part of the VOIT sequence was interpolated by trans- picture being transformed was n pixel square, 2n FFTs 
forming the picture as one big block rather than 16 are nece ssary , giving a total of 2n2 logi n clock periods 
small ones, and extracting the four principal vectors per 2-D FFT, 

firom the coirelation surface. The interpolated pictures Since such an arrangement could perform complex 
were almost indistinguishable from those obtained using 55 FFTs, it would be possible to arrange the data so that 
many blocks. However, this sequence is a rather special two real arrays could be transformed in one operation, 
case, since it only shows three principal objects (the In order to calculate the correlation surface for one 
gate, the car and the background), and none of these are block, 2 2-D FFTs are required, giving the total number 
moving very fast. A sequence containing many fast of clock periods as 2n^ logi n for the FFTs. 
moving objects or rotating objects would probably look 60 Consider a simple case where we want to measure 
much better if more blocks and hence vectors were velocities on a picture-by-picture basis. If the picture 

was transformed as one block of 512 elements square, 
3.2.3 SUMMARY OF MOTION VECTOR transfom could be performed in about 0.35 seconds 

ASSIGNMENT INVESTIGATIONS . squired transform rate 

. . xiwi^o would be 40 mS per picture at a 50 Hz field rate, so tiiat 

The experiments descnbed above showed that it was about 10 such 'FFT engines' would be needed. This 
possible to correctly assign motion vectors to pixels totals about 80 multipliers and 60 adders. Although this 
usmg the method described in section 2.2. The only represents a large amount of hardware, it is quite feasi- 
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ble. Additional hardware would be needed to multiply 
the phase arrays between the transforms, but this would 
onl y repr esent about l/9th of the hardware required for 
theFFT. 

Further hardware will be needed to perform the peak 5 
hunting and interpolation process. Such hardware 
would probably be similar in size to a digital filter with 
an aperture of 3 pixels square (as 9 points have to be 
examined to determine if a point is a maximum), but 
would be based around comparators rather than multi- 10 
pliers. Thus this would not require a significant amount 
of hardware. 

4.2 VECTOR ASSIGNMENT HARDWARE 

The complexity of this part of the equipment would 15 
again depend very much on the specific application. 
For something like a standards converter, a motion 
vector of sub-pixel accuracy would be required for each 
pixel, whereas in a bandwidth reduction system, it may 
only be necessary to assign vectors to pixels on a block- 20 
by-block iSasis, and sub-pixel accuracy may not be im- 
portant. The number of trial vectors per block would 
also directly affect the size of the hardware. 

Consider the worst case where a vector of sub-pixel 
accuracy is needed for each pixel. Hardware resembling 25 
two spatial interpolators, a subtractor, and a spatial 
filter would be required for each trial vector. This 
might represent something of the order of 20 to 30 
multipUers with associated adders. If we were trying 4 
vectors per pixel, a total of about 100 multipliers may be 30 
required. 

On the other hand, a block based vector assignment 
system in which sub-pixel accuracy was not important 
would be a lot simpler. The hardware required could 
reduce to as little as a few subtractors and adders per 35 
vector. 

If the transforms were carried out in smaller blocks, 
costs might be reduced shghtly, but the vector assign- 
ment hardware may become more complicated. If mo- 
tion vectors are not required to sub-pixel accuracy, it 40 
may be possible to get away with filtering the input 
picture by a fa ctor o f two in both directions prior to 
performing the FFTs, and this could reduce the cost of 
the hardware by a factor of 4. 

5,0 USING ONE DIMENSIONAL TRANSFORMS 

The first stage of the motion vector measurement 
technique as described involves performing a two-di- 
mensional phase correlation between two successive 
images in a scquency. For a picture portion that con- 50 
tains mXn elements, this involves carrying out m Fou- 
rier TransfomM of length n followed by n Fourier 
Transforms of length m, calculating the phase differ- 
ence for each of the (mXn) spatial frequencies, and 
performing the same number of reverse transforms. The 55 
resulting phase correlation surface is then interrogated 
to fmd the location of the dominant peaks. 

Most of this calculation can be avoided by carrying 
oat a pair of orthogonal one dimensional transforms on 
the picture data after applying a one dimensional filter 60 
to the picture, and hence measuring the components of 
the dominant shifts parallel to given axes. This could be 
described as a 'variables separable' approach. For exam- 
ple, the picture could first be vertically filtered by sum- 
ming all elements in each column of pixels. A phase 65 
correlation would then be performed between the re- 
sulting row of values and the row of values that resulted 
from filtering the previous picture. The dominant peaks 
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in the correlation function would give the horizontal 
components of the dominant motion vectors in the pic- 
ture. A similar process would be performed to measure 
the vertical components of the dominant shifts. 

This approach results in significandy fewer trans- 
forms being needed. For a picture portion of n by n 
samples, the two dimensional approach requires a total 
of 4n transforms, whereas the variables separable ap- 
proach requires 4. The disadvantage of this approach is 
that the peaks in the correlation function may be more 
subject to noise and more difficult to pick out. 

As it stands, this method cannot uniquely identify the 
locations of the dominant peaks, since there is no indica- 
tion as to which horizontal and vertical components 
shouW be paired up. This is only a problem when more 
than one dominant vector is of interest It can be shown 
that the location of n peaks can be uniquely identified by 
calculating at least n+ 1 correlations on mutually non- 
parallel axes. As n rises, however, the probability of 
there being more than n possible peak locations dimin- 
ishes even if there are n or less axes. In a practical case 
where we want to fmd, say, 4 peaks, 4 correlations 
would probably be adequate. The probability of assign- 
ing incorrect peak locations also decreases with the 
accuracy of the coordinate measurement. Even if some 
uncertainty in peak location remains, this can always be 
overcome by increasing the numbers of shift vectors 
that are tried in the vector assignment stage. 

To summarize, the amount of calculation required to 
find the dominant peaks in the correlation surface for 
two pictures can be significantiy reduced by performing 
a number of one dimensional correlations on the fUter 
picture: The filtering operation amounts to summing the 
picture elements along lines perpendicular to the axis of 
the correlation. The correlations must be performed on 
mutually non-parallel axes. To uniquely identify the 
locations of n peaks using this technique, it is necessary 
to carry out at least n-hl correlations, although forv 
n>3, fewer correlations will usually suffice. If 4 corre- 1 
lations were performed on a picture portion d pixels I 
square, the number of Fourier Transforms required per | 
input picture would be 8, compared to 4d for the full 
two dimensional method. For a typical block size, this 
amounts to savings in hardware of the order of a factor 
of 30. The penalty paid is that the peaks will probably 
be broader and noisier, and some false peak locations 
may be produced. 

6.0 APPLICATIONS 

This description has explained a novel technique for 
applying phase correlation to measure motion in televi- 
sion pictures. Computer simulations have shown that 
the technique appears to work very well (probably 
better than any other published technique). Various 
refinements to the technique have been suggested. 

The invention will be seen to lie in the combination of 
correlation of two pictures to determine correlation as a 
function of selective displacement (in the general case in 
two dimensions), thereby to determine a plurality of 
peak correlation values corresponding to respective 
motion vectors, followed by testing on a pixel-by-pixel 
or block-by-block basis which of these motion vectors 
gives the best fit in deriving one of the pictures from the 
other. 

The technique appears to offer great hope that the 
quality of equipment such as standards converters could 
be dramatically improved. In addition, it opens the door 
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to new applications such as the improvement of the 
motion portrayal of film. 

An advantageous application for this technique is to 
improve the quality of a motion adaptive bandwidth 
compression system. This involves measuring the prin- 5 
cipal motion vectors in a scene, and assigning them on a 
block-by-block basis (a block being about 6 pixels 
square). 

The motion vector measurement technique can be 
usefully applied in any situation where it is necessary to 10 
perform temporal interpolation on a series of images or 
follow the movements of objects m a scene. The follow- 
ing paragraphs describe some applications in more de- 
tail. 

6. 1 IMPROVEMENT OF FILM MOTION 
PORTRAYAL 

When a tilm is shown on television, each frame is 
displayed twice (once scanned as an odd television field 
and once as an even field in an interlaced system). Such 20 
an anangement enables a film shot at 25 (or 24) frames 
a second to be seen at (nearly) the correct speed. While 
the process of repeating pictures on successive fields 
significandy reduces the level of flicker that would be 
experienced if each picture was only shown once, the 25 
process also introduces modon impairments. As the 
observer's eye tracks moving objects, a double image 
appears on the retina because each moving object is not 
shown at the correct place at the right time. 

The motion impairments introduced by this system 30 
could be virtually eliminated if intermediate pictures in 
the film sequence were generated using motion com- 
pensated interpolation. 

Each pixel in the intermediate picture could be gener- 
ated by averaging the luminance levels at the corre- 35 
spending pixels in the following and preceding pictures, 
displaced by plus and minus half the appropriate motion 
vector respectively. For colour film, the values of the 
R, G and B components would each be derived in this 
way. 40 

Areas of uncovered or obscured background could 
not be dealt with in this way since such areas would not 
have a proper motion vector. In order to deal with these 
areas, it is first necessary to determine if the area corre- 
sponds to uncovered or obscured background. This can 45 
be determined by examining the component of motion 
vectors in the surrounding areas normal to the bound- 
ary. Areas of uncovered background will have neigh- 
bouring vectors pointing away from them, and ob- 
scured background areas will have neighbouring vec- 50 
tors pointing towards them. Picture information in un- 
covered areas must be taken from the following picture, 
and information in obscured areas must come from the 
precedmg picture. In order to determine the location of 
the required picture information in the following or 55 
preceding picture, it is necessary to examine the motion 
vectors measured between the following two pictures 
or the preceding two pictures. The required area will be 
that one which would originate from or end up at the 
unassigned area when its motion is extrapolated back- 60 
wards or forwards in time. 

These principles can be applied equally well to other 
field or film rates. For example, when showing film on 
a 60 Hz television system, two intermediate frames 
would be generated between each pair of film frames. 65 
New pictures would be generated 2/5 and 4/5 of the 
way between one pair of frames, and 1/5 and 3/5 of the 
way between the next pair, and so on. The generation of 
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an intermediate picture at a time other than halfway 
between two frames involves exactly the same process 
as outlined above except that vector assignment is per- 
formed at the appropriate time between the pictures 
rather than midway. The amplitude of the luminance 
(or colour component) signal for each pixel in the pic- 
ture being generated would be a weighted average of 
the value at the appropriate points in the adjacent pic- 
tures, such that a proportionally higher weighting is 
given to the value in the nearest picture. 

6.2 HIGH QUALITY STANDARDS 
CONVERSION 

Whenever it is necessary to convert pictures from 
one field rate to another, temporal interpolation is rcr 
quired. Knowledge of the motion vectors of the scene 
enables this process to be carried out with far fewer 
impairments that are introduced using conventional 
techniques. 

In order to perform motion compensated standards 
conversion it is first necessary to decode the signal into 
luminance and chrominance components if it was in a 
coded form, and convert the incoming interlaced pic- 
tures to a sequentially scanned standard. This latter 
process in itself can benefit from the knowledge of the 
scene's motion vectors, since one field can be displaced 
by the appropriate motion vector with respect to the 
other before interpolating the missing lines. Alterna- 
tively a vertical-temporal fdter can be used, although 
this will result in some loss of vertical detail in moving 
areas. 

Once the input pictures have been converted to se- 
quential form, motion compensated temporal interpola- 
tion is used to generate pictures at the times required by 
the output field rate. This process is identical to that 
described above for improving the motion portrayal of 
film, except that it is only necessary to generate every 
other line of the output pictures, assuming conversion 
to an interlaced standard. 

6.3 GENERATING OF SMOOTH SLOW MOTION 
SEQUENCES 

It is often necessary to replay a short sequence of 
television pictures in slow motion. Conventionally this 
is done by repeating each field several times (or repeat- 
ing a field followed by a field generated using a simple 
interpolation process). This results in jerky motion. It is 
possible to use a special camera with a very high field 
rate to achieve smooth slow motion, although this can 
introduce operational difficulties. 

Smooth slow motion could be achieved using con- 
ventional cameras if extra field could be generated using 
temporal interpolation. This can be done using the pro- 
cess described earlier for performing standards conver- 
sion. The number of intermediate fields that need to be 
generated would depend on the amount by which the 
motion was to be slowed down. 

The electronics required to generate the intermediate 
fields would be placed on the output of a conventional 
video tape recorder with simple slow motion playback 
facilities. The equipment would measure the number of 
times each field was repeated and generate the appro- 
priate number of intervening fields to replace the re- 
peated fields. 

In order to obtain sharp pictures it would be neces- 
sary to use a camera with a short integration time. This 
does not present any significant operational problem; 
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indeed future cameras may all tend to have short inte- 
gration times. 



6.7 INCREASING FIELD RATE 



. . -r^^^ ^^^^^^^^^^ Interpolation may be used to increase the field rate 

6.4 VTOEO NOISE REDUCTION for display purposJ. 

In order to reduce noise on television pictures, it is ^ I claim: 
necessary to apply a filter to the signal. Spatial filtering 1. A machine method of TV picture motion measure- 
results m loss of detail m the picture which is usually ment, comprising a first step of correlating first and 
unacceptable. Temporal filtering can be very effective, second pictures to determme correlation as a function of 
although straightforward temporal filtering mtroduces selective displacement, thereby to determine a plurality 
serious impairments in moving areas of the picture, so of peak correlation values corresponding to respective 
such filtering has to be disabled in these areas. This motion vectors, and a second step of testing for each of 
results in the appearance of noise in and around moving a plurality of elementary areas of the pictures which of 
objects. these motion vectors gives the best fit in deriving one of 
Motion vector measurement techniques can be incor- ^ the pictures from the other, and wherein said second 
porated in the temporal filtering process so that it may step comprises the following operations: 
be extended into moving areas. In the case of first order (a) an elementary area of a picture is derived from 
recursive temporal filtering, this would entail shifting said first picture using each of said motion vectors 
each pixel (or group of pixels) in the frame store by the determined in said first step, thereby to form a set 
corresponding motion vector prior to performing each 20 derived elementary areas corresponding to said 
recursion. If the filtering was performed using a trans- motion vectors respectively; 
versal filter, the readout address of each store would be (b) each derived elementary area is compared with 
displaced according to the sum of the local motion the corresponding elementary error as a sum over 
vectors over the number of fields between the output the elementary areas of the magnitudes of the difr 
filed and the stored field. 25 ferences between corresponding points of said ele* 

mentary areas; 
(c) that one of said motion vectors which results in 

Motion vector measurement can be used to great the smallest comparison error is selected as giving 

advantage in almost ail video bandwidth reduction the best fit in deriving said elementary area of said 

techniques since knowledge of localized motion vectors 30 second picture from said elementary area of said 

is the key to being able to e^loit the temporal redun- first picture; 

dancy in the signal. An adaptive subsampling system (d) repeating operations (a), (b) and (c) for each of 

could use knowledge of motion vectors to enable miss- said plurality of elementary areas. 

ing samples to be reconstructed from preceding fields. 2. A method according to claim 1, wherein the seiec- 

DFCM systems can use motion vectors as the basis of 35 tive displacement is in two dimensions and two-dimen- 

the predictive element In both these systems, informa- sional correlation is performed. 

tion on thie motion vectors would be derived at the 3, A method according to claim 1, wheran a plurality 

transmitter by performing the measurement between of one-dimensional correlations are employed using 

two successive fields, and could be transmitted to the one-dimensional displacements in different directions. 

receiver along with the rest of the signal. ^ 4. A method accordmg to claim 3, wherein the num- 

6.6 ADDING COLOUR TO BLACK AND WHITE ""^ll^^"^^^^^^^ 



6.5 BANDWIDTH REDUCTION 



FILMS 



a plurality of motion vectors are determined as those 
vectors which have components in the said different 
Recently a process known as *Colorization' has been directions matching the peak correlation values pro- 
developed (by Colorization Inc.) whereby colour ver- vided by the one*dimensional correlations, 
sions of black and white films can be produced. A still 5. A method according to claim 1, wherein the mo- 
frame firom the film (stored in a computer) is 'coloured tion vectors are measured to sub-pixel accuracy by 
in* by an artist. The computer then attempts to colour in interpolating the correlation function, 
the following frames by working out which areas are 6. A method according to claim \, wherein the step of 
stationary, and colouring these areas with the colour correlating is performed separately in respect of a plu- 
previously specified for this area. Moving areas are not rality of blocks of pixels and the whole assembly of 
dealt with and require the intervention of the artist. motion vectors thereby determined is used in the second 

If motion vectors were assigned to each point in a step, 
frame to be coloured in (by correlation with the previ- 7. A method according to claim 1, wherein the step of 

ous frame), and each point assigned the colour of the correlating is performed separately in respect of a plu- 

point in the previous frame pointed to by the vector, rality of blocks of pixels and, in the second step, only 

then the technique could be extended to moving areas. the motion vectors determined in relation to a given 
The only areas that would require the intervention of block are used in the testing in relation to that block, 
the artist would be areas of uncovered background and 5Q 8. A method according to claim 1, wherein the step of 

areas of complicated motion which could not be correlating is performed separately in respect of a plu- - 

tracked. (There would be no problem with obscured rality of blocks of pixels and, in that the second step, in 

background because the vectors are assigned directly to the testing in relation to a given block, the motion vec- 

the second of the two pictures, so no areas could ever be tors used are those determined in relation to that block 
flagged as 'obscured'.) Addition of motion vector mea- 65 and a plurality of adjacent blocks, 
surement to this process would thus greatly decrease 9. A method according to claim 6, 7 or 8 wherein 

the amoimt of work the artist has to do and so signifi* each block is windowed using a raised-cosine or other 

cantly speed up the colouring process. windowing aperture. 
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10. A method according to claims 6, 7 or 8^ wherein 
each block is windowed and each windowed block is 
surrounded with zeroes. 

11. A method according to claim 6, 7 or 8, wherein 
the blocks are overlapping blocks. ^ 

12. A method according to claim 1, wherein each 
elementary area consists of one pixel. 

13. A method according to claim 1, wherein each 
elementary area is a block of pixels. 

14. A method according to claim 1, wherein the cor- 
relation is phase correlation. 

15. A method according to claim 14, wherem a phase 
correlation function is derived from an inverse Fourier 
transformer of the product of Fourier transforms of the 15 
two pictures. 

16. A method according to claim 1, wherein the cor- 
relation function is subjected to temporal filtering be- 
fore the peak correlation values are determined, so as to 
reduce noise peaks. 20 

17. A method according to claim 16, wherein the 
temporal filtering is temporarily inhibited whenever 
there is a shot change. 

18. A method according to claim 1, wherein an error 
surface is derived in the second step for each motion 
vector and, for each pbcel, the motion vector whose 
error surface exhibits the smallest value is assigned to 
that pixel. 

19. A method according to claun 18, wherein no 30 
motion vector is assigned when the smallest value ex- 
ceeds a predetermined threshold value. 

20. A method according to claim 18, wherein each 
error surface is subjected to spatial filtering to reduce 
the effect of noise. 35 
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21. A method according to claim 18, wherein the 
error surfaces are multiplied by factors which increase 
with the vector lengths pertaining to the surfaces. 

22: A method according to claim 1, comprising the 
further steps of effecting temporal interpolation of the 
pictures utilizing the motion vectors to interpolate the 
positions of moving objects in accordance with the time 
point of the interpolated picture between the preceding 
and succeeding pictures from which it is received. 

23. A method according to claim 22, wherein the 
picture mformation for uncovered background, with 
motion vectors pointing away therefrom, is taken into 
the mterpolated picture from the succeeding picture 
and the picture information for obscured backgruond, 
with motion vectors pointing there-mto, is taken into 
the interpolated picture from the preceding picture. 

24. A method according to claim 22 or 23, wherein a 
plurality of pictures are interpolated between consecu- 
tive input pictures so as to generate a slow motion se- 
quence or increase the field rate for display purposes. 

23. A method according to claim 1, wherein the pic- 
tures are subjected to temporal fdtering to reduce noise 
and the filtering is extended to include the moving ob- 
jects shifted from field to field in accordance with their 
assigned motion vectors. 

26. A method according to claim 1, wherein the pic- 
tures are transmitted or recorded with bandwidth re- 
duction, with the inclusion of motion vector informa- 
tion for reconstruction of missing samples of moving 
objects. 

27. A method according to claim 1, wherein black 
and white pictures are coloured by automatically re- 
peating assigned colours from frame to frame at loca- 
tions determined by motion vectors. 
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